@spydon Please take a look at the failing test, I suspect it might be due to differences between Flutter versions on my machine and in the CI pipeline.

@spydon Please take a look at the failing test, I suspect it might be due to differences between Flutter versions on my machine and in the CI pipeline.

I would guess that it is due to the classic font rendering problem when using goldens:
https://pub.dev/packages/alchemist#about-platform-tests-vs-ci-tests

Golden tests involving texts are usually a pitfall because text rendering is influenced by the OS where the test is run, so if we generate a snapshot on linux and then try to run the test on MacOs or windows, there are high possibility that the test will fail.

So IMO we shouldn't use golden tests for these situations.

We can use them, but render boxes instead where the text would have been.
Alchemist for example has a mode for this.
(There are also a few fonts that are specialized for golden testing, but that's more work.)

Alchemist for example has a mode for this.
Interesting, would be cool to test this.

We can use them, but render boxes instead where the text would have been.

Flutter is already doing this for us: packages/flame/test/_goldens/text_box_component_test_1.png.

So the problem here must be something else. There are diff files generated by the test, but I can't see a way to access them from the CI. Could there be an option somewhere to include those files into the archive with the logs?

Another approach could be to run the test on another local machine with a different version of Flutter and see if the test succeeds or not.

I hope that after merging #1634 we can switch to sprite fonts here, and there wouldn't be a problem anymore.

Feels a bit like cheating though, since normal fonts aren't tested then (and this will most likely not be the last time we have this problem in the CI).

I looked at Alchemist documentation, and they say they use Ahem font when rendering in CI, which replaces all glyphs with blocks. However, Flutter itself uses Ahem font for rendering goldens, yet somehow we see a 1-pixel shift. I'm not sure using Alchemist would solve this...

There's also the golden_toolkit package, and the approach they took is to bundle Roboto font together with the tests, and load it directly from there: https://github.com/eBay/flutter_glove_box/blob/9e881929543e1ed634f854897ded903d4bc2f55a/packages/golden_toolkit/lib/src/font_loader.dart I wonder if we can maybe replicate that somehow?

I just noticed that all positions have .5 pixels, maybe this wouldn't cause anti-aliasing if the positions were in .0 instead?
(Just to keep the thread straight, this was already tried)

@st-pasha Let's put skip: true on the test and open an issue about fixing it, so that this feature can be included in v1.2.0

So, it looks like the issue is caused by different ways that the fonts are smoothed on different operating systems. I made these zoomed-in screenshots, which show that both OSes perform smoothing, but in a different way:

I've tried to render the text using a paint with antialiasing turned off - but it didn't make any difference, looks like macOS simply doesn't support switching off antialiasing for fonts (except maybe as a global system option, but that would be too disruptive for day-to-day life)...

In view of this, I believe the best we can do is to not rely on system font rendering altogether and use sprite fonts instead (at least for golden files); or wait until such time as Flutter team figures how to make golden tests match if the difference between images is up to anti-aliasing effects.

So, it looks like the issue is caused by different ways that the fonts are smoothed on different operating systems. I made these zoomed-in screenshots, which show that both OSes perform smoothing, but in a different way

That is the classic Flutter issue, didn't we post that one here? Otherwise I know @erickzanardo has the link to it somewhere, because he has a workaround for it in that issue.
Alchemist should solve that issue.

Here is the issue (Erick digged it up):
flutter/flutter#56383
(apparently the suggested workaround doesn't work all the time, so we should avoid that)

because he has a workaround for it in that issue.

The suggested "workaround" - to change the font smoothing at system global level - would only work on a computer that is dedicated solely to running golden tests and to nothing else. Because in a day-to-day life font smoothing is a very much desirable feature, there is a reason companies invest money in improving it.

Of course, ideally, Apple would have an API for rendering a specific string without any antialiasing. But apparently they don't have this, and it's not clear that they ever will (even if they do, we don't want tests that run on some version of OS but not on other, and not everyone can upgrade their OS easily).

In view of this, there may be no other option than manual sprite-based fonts. If Alchemist has a work-around, then this is what they must be doing.

The suggested "workaround" - to change the font smoothing at system global level - would only work on a computer that is dedicated solely to running golden tests and to nothing else. Because in a day-to-day life font smoothing is a very much desirable feature, there is a reason companies invest money in improving it.

Yeah, sorry I was unclear, I didn't mean that we should use that work around, it was just how I remembered the issue. :)

If Alchemist has a work-around, then this is what they must be doing.

Indeed, let's investigate that post v1.2.0.

The suggested "workaround" - to change the font smoothing at system global level - would only work on a computer that is dedicated solely to running golden tests and to nothing else. Because in a day-to-day life font smoothing is a very much desirable feature, there is a reason companies invest money in improving it.

Yeah, that workaround was something that used to work, but since some time isn't accurate anymore. There is only one workaround that I think is to be trusted. Running golden tests on the same environment as the CI will run them. On a multi platform world, docker would be an alternative for that, but that would render a very bad developer experience IMO.

In view of this, there may be no other option than manual sprite-based fonts. If Alchemist has a work-around, then this is what they must be doing.

I am a little bit skeptical about this still, giving the nature of the issue. But I would love to see this in action and would love even more to be proven wrong. Lets try to plan for doing this, even if as a POC for now.